Automated Benchmarking of KR-systems

Benchmarking is an important part of scientific work on solving techniques for KR systems. The implementation of hand-crafted scripts for each benchmark problem is cumbersome and repetitive. While most benchmarks are similar such that the process appears to be largely automatable, there are also differences which inhibit a complete reuse of existing scripts, e.g., different parameters to be measured and different aggregation functions to be applied. This calls for a tool which is applicable out of the box for a large range of benchmarks, but still allows for easy customization if needed. In this paper, we present such a system for automated benchmarking, which we base on a formalization of customizable benchmarks. The system captures the whole benchmarking process, including the run of individual instances, extraction of relevant information from the command outputs, aggregation of the results, and generation of the final benchmark table. A single command can then be used to generate the final table in LTEX format which can conveniently be copied and pasted into a paper. In contrast to existing approaches, which usually focus on standardized settings such as in large-scale competitions, ours focuses on the possibility for customization, i.e., on benchmarks with possibly heterogeneous parameters and values to be measured, such as those which arise when evaluating new evaluation techniques.